{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2110a2da",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Automatic Differentiation\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "130439cd",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:26:08.286501Z",
     "iopub.status.busy": "2023-08-18T19:26:08.285693Z",
     "iopub.status.idle": "2023-08-18T19:26:10.052257Z",
     "shell.execute_reply": "2023-08-18T19:26:10.050994Z"
    },
    "origin_pos": 3,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1b3f966",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Differentiating the function\n",
    "$y = 2\\mathbf{x}^{\\top}\\mathbf{x}$\n",
    "with respect to the column vector $\\mathbf{x}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4253cfab",
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "7"
    },
    "execution": {
     "iopub.execute_input": "2023-08-18T19:26:10.056833Z",
     "iopub.status.busy": "2023-08-18T19:26:10.055871Z",
     "iopub.status.idle": "2023-08-18T19:26:10.084858Z",
     "shell.execute_reply": "2023-08-18T19:26:10.083727Z"
    },
    "origin_pos": 8,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0., 1., 2., 3.])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.arange(4.0)\n",
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35a0803e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Before we calculate the gradient\n",
    "of $y$ with respect to $\\mathbf{x}$,\n",
    "we need a place to store it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2a001d1e",
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "9"
    },
    "execution": {
     "iopub.execute_input": "2023-08-18T19:26:10.088716Z",
     "iopub.status.busy": "2023-08-18T19:26:10.087816Z",
     "iopub.status.idle": "2023-08-18T19:26:10.092878Z",
     "shell.execute_reply": "2023-08-18T19:26:10.091740Z"
    },
    "origin_pos": 13,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "x.requires_grad_(True)\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34223f79",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "We now calculate our function of `x` and assign the result to `y`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "6e3bd777",
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "11"
    },
    "execution": {
     "iopub.execute_input": "2023-08-18T19:26:10.096336Z",
     "iopub.status.busy": "2023-08-18T19:26:10.095772Z",
     "iopub.status.idle": "2023-08-18T19:26:10.105236Z",
     "shell.execute_reply": "2023-08-18T19:26:10.104075Z"
    },
    "origin_pos": 17,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(28., grad_fn=<MulBackward0>)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = 2 * torch.dot(x, x)\n",
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2f52bd4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can now take the gradient of `y`\n",
    "with respect to `x`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "21b134ae",
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "12"
    },
    "execution": {
     "iopub.execute_input": "2023-08-18T19:26:10.108600Z",
     "iopub.status.busy": "2023-08-18T19:26:10.108011Z",
     "iopub.status.idle": "2023-08-18T19:26:10.160854Z",
     "shell.execute_reply": "2023-08-18T19:26:10.159702Z"
    },
    "origin_pos": 25,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 0.,  4.,  8., 12.])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.backward()\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74e7747a",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "We already know that the gradient of the function $y = 2\\mathbf{x}^{\\top}\\mathbf{x}$\n",
    "with respect to $\\mathbf{x}$ should be $4\\mathbf{x}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5030e37d",
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "14"
    },
    "execution": {
     "iopub.execute_input": "2023-08-18T19:26:10.164665Z",
     "iopub.status.busy": "2023-08-18T19:26:10.163930Z",
     "iopub.status.idle": "2023-08-18T19:26:10.171033Z",
     "shell.execute_reply": "2023-08-18T19:26:10.169923Z"
    },
    "origin_pos": 30,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([True, True, True, True])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.grad == 4 * x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e19d827",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now let's calculate \n",
    "another function of `x`\n",
    "and take its gradient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "add5cf4b",
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "20"
    },
    "execution": {
     "iopub.execute_input": "2023-08-18T19:26:10.174691Z",
     "iopub.status.busy": "2023-08-18T19:26:10.173957Z",
     "iopub.status.idle": "2023-08-18T19:26:10.181847Z",
     "shell.execute_reply": "2023-08-18T19:26:10.180759Z"
    },
    "origin_pos": 37,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1., 1., 1., 1.])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.grad.zero_()\n",
    "y = x.sum()\n",
    "y.backward()\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32049daa",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Sum up the gradients\n",
    "computed individually for each example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "1baa40bd",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:26:10.185096Z",
     "iopub.status.busy": "2023-08-18T19:26:10.184685Z",
     "iopub.status.idle": "2023-08-18T19:26:10.192537Z",
     "shell.execute_reply": "2023-08-18T19:26:10.191435Z"
    },
    "origin_pos": 45,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0., 2., 4., 6.])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.grad.zero_()\n",
    "y = x * x\n",
    "y.backward(gradient=torch.ones(len(y)))\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e47c4826",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Move some calculations\n",
    "outside of the recorded computational graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "107ac041",
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "21"
    },
    "execution": {
     "iopub.execute_input": "2023-08-18T19:26:10.196001Z",
     "iopub.status.busy": "2023-08-18T19:26:10.195456Z",
     "iopub.status.idle": "2023-08-18T19:26:10.203246Z",
     "shell.execute_reply": "2023-08-18T19:26:10.202155Z"
    },
    "origin_pos": 50,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([True, True, True, True])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.grad.zero_()\n",
    "y = x * x\n",
    "u = y.detach()\n",
    "z = u * x\n",
    "\n",
    "z.sum().backward()\n",
    "x.grad == u"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "cb8c674b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:26:10.206880Z",
     "iopub.status.busy": "2023-08-18T19:26:10.206001Z",
     "iopub.status.idle": "2023-08-18T19:26:10.213592Z",
     "shell.execute_reply": "2023-08-18T19:26:10.212476Z"
    },
    "origin_pos": 55,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([True, True, True, True])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.grad.zero_()\n",
    "y.sum().backward()\n",
    "x.grad == 2 * x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe357b0b",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Even if\n",
    "a function required passing through a maze of Python control flow\n",
    "we can still calculate the gradient of the resulting variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "ab14ef91",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:26:10.237298Z",
     "iopub.status.busy": "2023-08-18T19:26:10.236886Z",
     "iopub.status.idle": "2023-08-18T19:26:10.243577Z",
     "shell.execute_reply": "2023-08-18T19:26:10.242480Z"
    },
    "origin_pos": 70,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(True)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def f(a):\n",
    "    b = a * 2\n",
    "    while b.norm() < 1000:\n",
    "        b = b * 2\n",
    "    if b.sum() > 0:\n",
    "        c = b\n",
    "    else:\n",
    "        c = 100 * b\n",
    "    return c\n",
    "\n",
    "a = torch.randn(size=(), requires_grad=True)\n",
    "d = f(a)\n",
    "d.backward()\n",
    "\n",
    "a.grad == d / a"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "language_info": {
   "name": "python"
  },
  "required_libs": [],
  "rise": {
   "autolaunch": true,
   "enable_chalkboard": true,
   "overlay": "<div class='my-top-right'><img height=80px src='http://d2l.ai/_static/logo-with-text.png'/></div><div class='my-top-left'></div>",
   "scroll": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}